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ABSTRACT 



This 1999 observational study provided a school district 
with clear information about the experience of immigrant students in two 
elementary English- as -a- Second- Language (ESL) models, giving administrators, 
teachers, and school board members a window into the daily classroom life of 
ESL students. This program evaluation involving observational research design 
was implemented in a Wheaton, Illinois, school district. An innovative, 
self-contained elementary ESL program located at two of the district *s 
elementary schools was evaluated through a matched-pairs research design that 
compared the experience of students in this program with the experience of 
similar students in the district *s ESL pull-out programs. This report 
describes the observational research methodology and the results . A 
statistical analysis found significant differences between the level of 
engagement and teacher student interaction in the two program models, with 
the results favoring the experimental pod of the ESL program. These results 
are illustrated with qualitative data, collected through ethnographic 
narratives, which are presented in the report in the form of case study 
comparisons. Six appendices are included, containing a glossary of terms, 
detailed displays of data, sample forms, and other background information 
regarding the research methodology used. (Contains 10 references.) (KFT) 
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Abstract 



In 1999, an ESL program evaluation involving an observational research 
design was implemented in Community Unit School District 200, Wheaton, 
Illinois. An innovative self-contained elementary ESL program located at 
two of the district’s elementary schools was evaluated through a matched 
pairs research design which compared the experiences of students in this 
program with the experiences of similar students in the district’s ESL pull- 
out programs. This report describes the observational research methodology, 
which was conducted by Wheaton College graduate students under supervision 
of a professor, and the results, which were presented to the school board and 
administrators in the district. A statistical analysis (through a paired sample 
t-test) found significant differences between the levels of engagement (or 
time on task) and teacher-student interaction in the two program models, with 
the results favoring the experimental “pod” ESL program model. These results 
were illustrated by the qualitative data, collected through ethnographic narratives, 
which were presented in the report in the form of case study comparisons. 



I. Background. 

This report presents an evaluation focusing on a model English as a second language 
(ESL) program (described here as the “pilot pod” program) which was implemented at the 
elementary level in District 200 during the 1998-99 school year and is continuing during 1999- 
2000. Funding for the pilot ESL program was approved in April 1998 with the expectation that 
this program will be evaluated on an ongoing basis and, if successful, consideration would be 
given to expanding the program in future years. 

The pilot program involved clustering students at two schools, Washington Elementary 
(grades K-I) and Madison Elementary (grades 2/3 and 4/5), in addition to the pull-out ESL 
programs already in existence at seven of the district’s elementary schools. The ESL students in 
the pilot program were tested at English language skill levels of I, 2, and 3, indicating that they 
needed a significant degree of support. 

To encourage integration with the standard curriculum and interaction with students in 
the regular classes, each ESL class in the pilot pod program was team-taught by a regular 
classroom teacher and a trained ESL teacher. In addition, each ESL class was connected with a 
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“buddy” classroom, which allowed the ESL students to participate in learning activities along 
with native English speaking children. The student/ teacher ratio was intentionally lowered in 
the pilot program in order to facilitate student progress. 

How successful was the pilot program? There are a number of ways to evaluate a new 
program, including test scores, portfolios of student work, and parent/student surveys. One 
particularly salient area to explore is the academic and social experience of students in the 
classroom. In other words, are ESL students in the pilot program consistently engaged in 
meaningful learning activities, and is their level of activity significantly higher than one would 
find in a typical pull-out program where the ESL student is mainstreamed during most of the 
day? 

To examine the academic and social experience of students in the pilot program, a 
qualitative research study was devised by the members of the ESL task force in collaboration 
with a professor and graduate students at Wheaton College. Phase One of this study was 
completed in April 1999, and the results of this phase were reported to the Board in June 1999. 
Phase Two, which was an expansion of Phase One, was completed in late September and early 
October 1999. The basic question addressed in both phases of the research is the following: 
With the students in the pull-out programs serving as a baseline, are the students receiving the 
additional services in the pilot programs, to a significant degree, more engaged in the learning 
process? 



II. Support from Prior Research. 

Longitudinal Research by Cummins, Collier, and Thomas. It is important to note that the 
pilot pod model adopted by District 200 is similar to ESL programs which have been described 
as effective in several widely-known research studies. 

Research by James Cummins (1981, 1984) in Canada in the 1970s and 80s established 
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two categories for second language acquisition (BICS and CALP) which have different rates of 
acquisition in educational contexts where students are immersed in a second language. BICS 
(Basic Interpersonal Communication Skills) refers to social and functional uses of a foreign 
language in everyday conversation, in contrast to CALP (Cognitive Academic Language 
Proficiency), which refers to the more complex language used in schooling. 

Drawing upon longitudinal data involving standardized academic test scores, Cummins 
contends that in an immersion context, young children tend to acquire a native-speaker level of 
proficiency in BICS in 1-2 years, while a grade-level proficiency in CALP may take as long as 4- 
7 years. He acknowledges that for older children prior educational skills and content can transfer 
into a new language, accelerating the process of acquiring CALP. For both primary and upper 
elementary children, however, Cummins’ research points to the need for high-quality ESL 
programs which combine language with academic content in order to increase the rate of 
acquisition of CALP, the language of schooling. 

Virginia Collier and Wayne Thomas (1997, 1999a) extended Cummins’ research through 
a decade-long study of immigrant and refugee students in school districts throughout the U.S. 
Their research examined the question of which ESIVbilingual program models were most 
effective in helping ESL students reach grade-level performance in school. Focusing on 
immigrant children who began their U.S. schooling in kindergarten. Collier and Thomas tracked 
student progress through standardized test scores and performance assessments to see how long it 
took for their average levels of academic performance to reach the average of native-English- 
speaking students at the same grade level. In their research, models which foster collaboration 
between ESL and grade-level teachers were shown to have a greater positive effect on immigrant 
student achievement than programs, such as pull-out and self-contained ESL, which deal with 
language and content in isolation. 

In a recently-published summary of this research. Collier and Thomas (1999b) describe 
an “enrichment” model which emphasizes collaboration between language and grade-level 
teachers: 
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Equal team teaching in an inclusion classroom (a bilingual/ESL certified specialist 
teaming with a mainstream certified teacher) has the potential to become an 
enrichment model. In contrast to remediation, enrichment adds to what the students 
already know. The strengths that English language learners bring to the classroom, 
including knowledge and life experiences from other cultural contexts, as well as a 
native speaker’s knowledge of another language, are used as resources for learning, 
as essential building blocks. In enrichment classes, students know that they are 
being challenged and are deeply engaged in the learning process. (1) 

Although the “enrichment” model Collier and Thomas recommend includes instruction in the 
students’ native languages, it bears a strong resemblance to the pilot pod programs adopted by 
District 200, which includes collaboration between ESL specialists and mainstream teachers in 
both the self-contained ESL classroom and with the “buddy” classroom. 



III. Comparison of Pull-Out and Pilot Pod Models Through Classroom Observational 
Research. 

A. Methodology. The evaluation study conducted by Wheaton College graduate students 
examined the educational experiences of students in pilot pod and pull-out programs who were 
similar in ethnic background, length of time in the school system, and grade level. Students who 
were typical of a particular grade level in the pilot program were selected, and each student was 
paired with a student in a pull-out program who was similar in terms of age, educational 
background, gender, English proficiency level, and length of time in the U.S. In Phase I (April 
1999), six pairs of students were selected and observed, for a total of 12 students ranging from 
first through fifth grade. During Phase II (September-October 1999), twelve pairs of students 
were selected, for a total of 24 students ranging from kindergarten through fifth grade. Informed 
consent was obtained from the families of all of the participants in the study. 
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The 36 ESL students selected for this study represented a broad range of ethnic 
backgrounds. Fifteen nationalities and twelve languages were represented in the sample: 



Table 1 

Countries of Origin 



Mexico 


10 


India 


1 


Vietnam 


6 


Cambodia 


1 


Bosnia 


5 


Chile 


1 


Russia 


2 


Iran 


1 


Venezuela 


2 


Croatia 


1 


Korea 


2 


Czech Rep. 


1 


China 


1 


Norway 


1 


Colombia 


1 







The systematic observation of these students totaled more than 162 hours, or approximately 4.5 
hours per student in their classroom contexts. 

The observers were education majors and prospective ESL teachers enrolled in graduate 
courses at Wheaton College. None of the observers had taught in District 200 before, nor had 
any attended District 200 schools. The researchers were given training and strict protocols for 
using the observation forms. Each observed a student in the pilot ESL program for three 90- 
minute sessions during a one-week period, followed by an identical observation schedule with 
the paired student in the pull-out program during the following week. The same observational 
methodology was used in both phases of the study. 

The observations included (1) a practice session in the classroom, which was not 
included in the final data analysis, (2) a 90-minute ethnographic narrative of each student’s 
experience in the classroom, (3) two 90-minute records o/each student’s “time on task, ” 
teacher- student interactions, and student-student interactions. (For a definition of each of these 
terms, see the addendum to this report.) The data were recorded on forms which were submitted 
to Dr. Alan Seaman at the graduate school for analysis. In addition, each researcher wrote a 10- 
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page analysis of the students that he or she had observed. Formal data analysis included 
tabulations of the data and comparisons of group means using statistical analysis, with the level 
of significance set at p<.05. 

This research design combined two types of data collection: classroom ethnography and 
observational research. By gathering data systematically through two different approaches, we 
hoped to support any findings through a process of triangulation. This approach follows the 
educational research procedures described in widely-used texts (Borg & Gall 1989; Nunan 1992; 
Alderson & Beretta 1992). The dual focus on interaction and time on task (also referred to as 
“work involvement” or “engagement”) reflects two lines of classroom research which are 
particularly relevant to the experience of ESL students (Chaudron 1988; Whitrock 1986, pp. 400- 
402). 

B. Limitations. As is common in qualitative research, this study used purposive sampling 
rather than a random sampling design. However, the sample size (18 pairs, 36 students) is large 
enough to suggest patterns which exist within and between the two programs. Although the pairs 
of students were matched as closely as possible, it was difficult to achieve perfect matches, and 
factors such as personality differences between the two students in each pair (e.g., introversion/ 
extroversion) could not be controlled. 



IV. Results of the Observational Study. 

The quantitative and qualitative evidence from the two phases of this study indicates that 
the experience of the students in the pilot ESL programs is different from that of the students in 
the pull-out ESL programs. An extensive summary of the data from both phases of the study is 
presented in Table 4 at the end of this report. 

In 16 of the 18 pairs studied, the student in the pilot program had a higher level of on-task 
activity (engagement) than the similar student in a pull-out program (See Tables 3A and B in 



Appendix B). Overall, students in the pilot pod programs were engaged in on-task behavior 94 
percent of the time that they were in class, while students who were mainstreamed in the pull-out 
programs were on-task an average of 85 percent of the time. 

Was this difference significant? Using a paired sample t-test for statistical analysis, the 
results for percentage of time on-task did indeed prove to favor the pilot program at a strong .002 
level of significance (See Table 2 below). The differences between the two programs in terms 
of the percentage of the time that the students were engaged in learning were unlikely to have 
happened by chance. This means that the students in the pilot pod program were consistently 
focused on learning during a significantly higher percentage of the classtime observed. 

These results point to the generalization that the students in the pilot program were, in 
each pair and overall, more active learners in the classroom than were their peers in the pull-out 
programs. They were more engaged and focused upon the activities set by the teacher. As a 
result, it is likely that the students in the pilot program were having a significantly different 
educational experience than they would have had if they were mainstreamed during most of the 
school day. 

It should be noted that this study reflected very positively on the work of the pull-out ESL 
teachers in each school. During their 30 minutes with the students each day, these teachers were 
observed providing an active, low-stress environment for learning similar to the environment in 
the pilot pod program. The differences between the two groups were primarily related to the 
ESL students’ experiences in regular classrooms while they were being mainstreamed. 

Although the overall frequency and rate of interaction (between teacher and student and 
student and peers) was higher for the students in the pilot pod program than for the students in 
the pull-out ESL programs, these differences between the two program models were not found to 
be significant. However, when the 

data collected during the 30-minute pull-out ESL class is separated from the mainstream class 
data, a very significant difference emerges. 

The results in Table 2 show that mainstream teachers were interacting with the ESL 
students at a much lower rate (.18 interactions per minute) than the rate of teacher-student 



interaction in the pilot pod program (.36 interactions per minute). Although the students’ time 
in pull-out ESL classes constituted only about 16 % of the total observation time, the pull-out 
teachers’ interactions with the students accounted for over 53 % of the total. In other words, a 
number of the students in the pull-out model were receiving very limited amounts of teacher- 
student interaction while they were mainstreamed during most of the day, with an intensive 
amount of interaction while they were in the 30-minute pull-out ESL classes. 

It is important to note that in approximately half of the pairs (nine out of the eighteen), 
the differences between the two programs were minimal, at least in terms of the constructs of 
engagement and interaction. In the other half, the differences were more dramatic. As we noted 
in the previous report, this suggests that some mainstream teachers are effective in engaging ESL 
students in their class activities — even students who are at the lower levels of English 
proficiency. 



Table 2: Significant Differences (Paired Sample T-Test): 



Variable 


Mean for 
Pilot Pod 


Mean for 
Mainstream 
and Pull-Out 


Difference 
in Means 


Standard 

Deviation 


t 

value 


Significance 

Level 

P< 


On-Task Percentage 


.9422 


.8556 


.0866 


.1004 


3.66 


.002 




Mean for 
Pilot Pod 


Mean for 
Mainstream 
(Without Pull- 
Out) 


Difference 
In Means 


Standard 

Deviation 


t 

value 


Significance 

Level 


On-Task Percentage 


.9422 


.8450 


.0972 


.1115 


3.69 


.002 


Teacher-Student 
Interaction Rate 
(interactions/minute) 


.3567 


.1814 


.1753 


.1906 


3.90 


.001 



This diversity of teaching approaches may, in part, account for the much broader ranges 
of engagement and interaction for the students in the pull-out programs than for the students in 
the pilot pod programs. In the pilot pod programs, the percentage of time on task ranged from 
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1.00 to .84, with nearly all of the students above the .90 level, while for the students in the pull- 
out programs the percentage extended from .97 to .59, with only half of the students at .90 or 
above. 



V. Specific Factors Noted by the Observers. 

What accounts for the success of the pilot pod program in encouraging a high level of 
engagement in learning activities? Comments by the researchers in their individual reports shed 
some light on this question. 

♦ All of the researchers noted the lower student/teacher ratio in the pilot pod classrooms. 
The ESL students who were mainstreamed in the pull-out programs had a low 
student/teacher ratio while they were in the ESL pull-out classrooms for 30 minutes each 
day, but during the rest of their school day, they were in classrooms with a ratio of 25:1 or 
more. As expected, the students in the pilot pod programs were receiving more 
individual attention from their teachers. 

♦ Another consistent theme in the researchers’ comments was that the ESL students in the 
pilot programs seemed less isolated than their peers in the pull-out programs. 
Mainstreamed in large classrooms with few peers from similar ethnic backgrounds, the 
ESL students in the pull-out programs simply had fewer opportunities to communicate 
with others and participate in the curriculum. In several cases, the regular teacher 
expressed frustration that he or she was unable to give ESL students much attention. 

♦ Finally, the researchers consistently noted that the experience of the students in the pilot 
pod programs was less fragmented than that of their peers in the pull-out programs. The 
mainstream classes were disrupted as students were pulled out for special purposes, and 
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the ESL students were expected to catch up on what the class had been covering during 
the pull-out period. The discontinuity of the students’ experiences in these classes 
seemed different from the clearly-structured, low-stress environment of the pilot pod 
classes. 

As one researcher noted, summing up the contrasts seen by most of the observers in this study: 

the ESL students [in the pilot pod program] seemed to be able to go more in depth 
with their activities because they were all together, working at a similar pace, and 
had longer time periods to devote to certain activities. The feeling [in the pull-out 
program] was that the students had disruptive, choppy days where they were 
involved in two classrooms with different field-trips, activities, and teachers. The 
pod teacher stated that her classroom works at a slower pace so that her students 
can learn important social and linguistic rules. She focuses on more basic, hands- 
on types of activities as well. From my observations, however, I would argue that 
the pilot pod classroom was in no way slower than either the mainstream classroom 
or the ESL classroom [at the other school]. 

VI. A Case-Study Comparison 

To sum up, a brief comparison may be useful in highlighting differences in the 
experiences of students in the two programs: 

Alexander' is a second grader who emigrated to the U.S. in the spring of 1998. He is 
described as a pleasant but shy boy who tends to be quiet in his classes in the pod program at 
Madison Elementary. Alexander still primarily speaks Russian at home and has been making 



'pseudonyms are used in this section to protect the identity of individual students. 



continuous but slow progress with acquiring oral proficiency in English. In the pilot pod 
program, he appears relaxed and focused as he works with other children in groups. 

The researcher describes Alexander’s context as follows: “The students almost always 
work in groups with one of the instructors. The atmosphere in class is relaxed and it’s obvious 
that students are familiar with the structure and routine of moving from group to group.” Every 
minute of class is used efficiently. During the course of an observation, Alexander meets with a 
reading group with six other students at his level of English proficiency. They work with a 
teacher’s aide on a book entitled My Garden, and Alexander seems to feel a sense of 
achievement when he reads a section the book aloud to the others. The vocabulary of the book 
is reinforced with vocabulary cards. Later, Alexander meets with a third-grade “buddy” reader 
who reads him a different book. He moves on to the computer lab, where he enters his name 
and begins to type out a writing exercise on the keyboard. 

Like Alexander, Lars is a second-grader who is at a beginning level of English 
proficiency and has come from Northern Europe. He speaks a Scandinavian language with his 
family at home. In the elementary school where he is in a pull-out program, Lars is quiet and 
attentive. He has difficulty speaking with his more fluent peers, even when in the pull-out ESL 
class. The content of his pull-out ESL class is coordinated with the second grade class in which 
he is mainstreamed - a unit on the theme of “community.” After an hour of instruction in the 
ESL pull-out class, Lars enters his grade-level class partway through a hands-on science activity 
related to donut production. The teacher directs Lars and two other ESL students to work on a 
computer in the back of the room while the other children complete the activity in groups. 

When he is able to participate with his classmates on an art project, his quietness 
evaporates for a moment to reveal a clever, mischievous personality. But faced with a literacy- 
related worksheet, Lars struggles to understand the instructions, staring at the paper, then looking 
around. After several minutes, the teacher circulates to his desk and reiterates the instructions in 
a form he can understand. 



The above description highlights some of the differences between the two program 



models that were repeatedly observed. Both students had similar language and socio-economic 
backgrounds, and they were at a similar level of English proficiency. In the pilot pod program, 
the content was carefully sequenced and modified in order to be comprehensible to the ESL 
students. On the other hand, Lars’ mainstream teacher (who was by all accounts, a skilled, 
caring teacher) was accustomed to assuming that the students could handle activities and 
exercises which involved context-reduced uses of language. 

Like many ESL students in these classes, Lars was seen either waiting patiently for an 
explanation of the instructions, and at times, he was isolated from group activities covered in 
class. His lower percentage of “time on task” during the lessons means that over the course of a 
year, he may have significantly less actual learning time than Alexander. By maximizing the 
amount of time ESL students spend on academic tasks, and using comprehensible language 
tailored to their levels of proficiency, the teachers in the pilot pod program are positioned to help 
these children close the gap between themselves and their peers as efficiently as possible. 



VII. Concluding Comments. 

This observational study provided the school district with clear information about the 
experience of immigrant students in the two elementary ESL models. This information gave 
administrators, teachers, school board members a window into the daily classroom life of ESL 
students. It will help the district make an informed decision about what models may be 
educationally valuable in the future. 

Other school districts may consider using a similar process when making decisions about 
the future. In addition to test scores, surveys, and other sources of information, observational 
data, if gathered and analyzed systematically, can be a helpful part of the evaluation process. 
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Appendix A 

Glossary of Terms Used in This Report 

Activity - An activity is defined as any block of time in the lesson plan. It can include lecturing 
by the teacher, pair work, worksheets, silent reading, science experiments, and so forth. 

Time on Task - Time on task (also known as “engagement” or “work involvement”) is defined 
as any period of time in which the student is focused on the activity introduced by the teacher. If 
the student is focused on something else (even if it is productive, such as reading), he/she is “off 
task..” In this study, time on task was recorded in terms of duration. 

Interaction - Interaction is defined as any time the student speaks to a teacher or aide, or to 
another student. This is recorded in terms of frequency of utterances rather than utterance length. 
If the teacher asks the student three questions in a row, and the student answers each question, 
then three interactions are recorded. If the ESL student is involved in a conversation with 
another student, one interaction is recorded for every time the ESL student speaks (i.e., for each 
“turn” in the conversation). 

Significance - Statistical significance is a computation of the probability that observed 
differences are the result of chance alone. A significant difference (between two means, for 
instance) at the .05 level implies that there is a 95% probability that the results are not the result 
of chance. (This probability level was set prior to the data gathering phase of the study.) It is 
important to note that in this qualitative study, which did not involve random sampling, the 
results are significant only for the group of students studied. The results are only suggestive of 
differences in the larger populations of the two types of ESL programs. 
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Appendix B 

Table 3A - Comparison of Student Experience in Pilot Pod and Pull-Out ESL Programs 

A. Pilot Pod Programs 

Case^ School/Grade Ethnic On Task Teacher-Student Student-Student 

Interaction Interaction 

(Woof total time) (frequency and interactions per minute) 

PHASE ONE (April 1999) 



lA 


Washington/ 1 


Bosnian 


.98 


101 (.55) 


257 (1.40) 


2A 


Washington/ 1 


Venezuelan 


.96 


76 (.46) 


62 


(.38) 


3A 


Madison/2 


Mexican 


.95 


79 (.43) 


194 


(1.07) 


4A 


Madison/2 


Bosnian 


.91 


88 (.49) 


66 


(.37) 


5A 


Madison/3 


Venezuelan 


.87 


77 (.43) 


70 


(.39) 


6A 


Madison/5 


Mexican 


1.00 


34 (.19) 


76 


(.42) 






PHASE TWO (September-October 1999) 






7A 


Washington/K 


Cambodian 


.98 


60 (.33) 


50 


(.27) 


8A 


Washington/K 


Mexican 


.93 


111 (.61) 


80 


(.44) 


9A 


Washington/ 1 


Colombian 


.93 


44 (.24) 


8 


(.04) 


lOA 


Washington/ 1 


Mexican 


.96 


59 (.32) 


93 


(.51) 


llA 


Madison/2 


Chilean 


.93 


72 (.40) 


43 


(.24) 


12A 


Madison/2 


Russian 


.96 


47 (.26) 


36 


(.20) 


13A 


Madison/3 


Iranian 


.84 


25 (.14) 


39 


(.22) 


14A 


Madison/3 


Vietnamese 


.97 


84 (.47) 


92 


(.51) 


15A 


Madison/4 


Vietnamese 


.96 


30 (.17) 


25 


(.14) 


16A 


Madison/4 


Russian 


.96 


76 (.42) 


69 


(.38) 


17A 


Madison/5 


Korean 


.92 


52 (.29) 


36 


(.20) 


18A 


Madison/5 


Croatian 


.95 


40 (.22) 


53 


(.29) 


Totals (means): 






.94 


64 (.36) 


75 


(.41) 



^ The case numbers reflect pairs of students: 1 A and IB were paired, 2 A and 2B, and so on. 
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BEST COPY AVAILABLE 



Table 3B 

B, Pull-Out Programs 



Case 


School/Grade 


Ethnic On Task 


Teacher-Student 


Student-Student 










Interaction 


Interaction 






(% of total time) 


(frequency and interactions per minute) 






PHASE ONE (April 1999) 




IB 


Hawthorne/ 1 


Vietnamese 


.77 


70 (.38) 


68 (.37) 


2B 


Hawthorne/ 1 


Mexican 


.89 


2 (.01) 


37 (.26) 


3B 


Longfellow/2 


Mexican 


.91 


112 (.62) 


16 (.09) 


4B 


Hawthorne/2 


Bosnian 


.90 


47 (.26) 


45 (.25) 


5B 


Lincoln/2 


Chinese 


.58 


1 (.01) 


6 (.03) 


6B 


Johnson/4 


Mexican 


.90 


56 (.31) 


47 (.26) 






PHASE TWO (Sept.-Oct. 1999) 




7B 


Johnson/K 


Korean 


.96 


46 (.25) 


134 (.74) 


8B 


Longfellow/K 


Mexican 


.87 


74 (.41) 


19 (.10) 


9B 


Longfellow/1 


Mexican 


.63 


2 (.01) 


6 (.03) 


lOB 


Lincoln/1 


Mexican 


.95 


36 (.20) 


97 (.54) 


IIB 


Longfellow/2 


Vietnamese 


.96 


62 (.34) 


29 (.16) 


12B 


Lowell/2 


Norwegian 


.77 


69 (.38) 


57 (.32) 


13B 


Longfellow/3 


Czech 


.79 


29 (.16) 


5 (.03) 


14B 


Lincoln/2 


Vietnamese 


.82 


54 (.30) 


64 (.36) 


15B 


Lincoln/4 


Vietnamese 


.92 


93 (.52) 


106 (.59) 


16B 


Lincoln/4 


Bosnian 


.95 


229(1.27) 


115 (.64) 


17B 


Lowell/5 


Indian 


.86 


5 (.03) 


18 (.10) 


18B 


Longfellow/5 


Bosnian 


.97 


56 (.31) 


32 (.18) 


Totals (means): 




.86 


58 (.32) 


50 (.28) 


Without ESL Pull-Out 




.84 


27 (.18) 


44 (.30) 
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Appendix C 



Systematic Observational Research Process 

I. Goal. As objectively as possible, to make comparisons between pull-out ESL 
students and pod-program students in terms of their experience in the academic 
setting. Using the pull-out programs as a baseline, is there a significant difference 
in the experience of these students to justify the additional cost? 

II. Procedures. 

A. Sample. Match pairs of students according to age, grade level, cultural 
background language background, gender, length of time in program and LAS 
scores. 

B. Develop Observation Forms. These will record classroom behaviors in terms 
of (1) an overall ethnographic narrative, (2) engagement or time on task, (3) 
teacher-student interaction, (4) interaction with peers. 

C. Select and Train Observers. Observers will have at least one hour of training 
in how to use the forms and one hour of practice in a classroom setting, with 
feedback, before they begin to collect the data. 

D. Obtain Consent. The purpose of the research will be communicated to 
principals, teachers, students, and parents. Informed consent will be obtained 
before any observations are done. Anonymity of students and teachers will be 
assured. 

E. Data Collection. Observers will shadow the ESL students during similar parts 
of the school day. During the first week, the student in the experimental program 
will be observed; during the second week the paired student in the pull-out program 
will be observed. After a practice observation using the forms, the observers will 
complete three 90-minute observations. The first will record an ethnographic 
narrative. The second and third will collect time-on-task and interaction data. 
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F. Data Analysis. The frequency counts for each behavior will be tabulated, and 
a paired samples t test (p < .05 level of significance) will be used to compare the 
data in the two groups. 

G. Reporting the Study. The report will include the statistical data and a 
summary of the insights from the data. Typical students will be selected to be 
profiled in two case studies. 
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Appendix D 



ESL Pilot Pod Evaluation Study 
Community Unit District 200 Public Schools 

OBSERVATION FORM 1 
ETHNOGRAPHIC NARRATIVE 

Teacher: _ CS-7W(»/< -fiyw.) Date: 

Location: Times: 

Observer: 



Instructions: Using complete sentences, write a narrative of the ESL students’ involvement 
in class over a one- to two-hour period. Include a diagram of the classroom which shows 
where the students are seated. Describe clearly the students’ involvement in tasks and 
activities. 



1 0 



/o; /o 



/s- ci ou ^ 

tru -!? f>f'^ 

fCUrfryUr 

ES“L UJOvk <^yJ7-€,i4y ^ 

fC(Ai~x Loa./^^ 

wLaf ll eto " 'T'kit 

*f(j Cl hill ^ 



c f>yvj 
C«\ishrJch'i{ia ^ 
cctn^LoctrJl 



J%u. C/)vv\f(t'fkA^ii^ A * hts 

kr f^u 

d.'^a.w Ljfl-* /I • hits 

^ \i\^ ^fde cf roevu wctLa^ 
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Appendix E 



ESL Pilot Pod Evaluation Study 
Community Unit District 200 Public Schools 

OBSERVATION FORM 2 
DESCRIPTIVE OBSERVATION RECORD 

Teacher' -fivKH ) Date. 

Location: Times: 

Observer: 



In the left column, record the activities that the student is engaged in over a two-hour period. 
In the central column, record the length of time that the student is “on task,” engaged in the 
activity. Record the specific times in terms of minutes and seconds. In the right columns, 
record the frequency of interaction between (I) the student and a teacher or aide and between 
(2) the student and other students. 



ACTIVITY ON-TASK BEHAVIOR INTERACTION INTERACTION 




Appendix F 



ESL Pilot Pod Evaluation Study 
Community Unit District 200 Public Schools 

INSTRUCTIONS FOR OBSERVERS 



1. In addition to these instructions, this packet contains (1) an assignment sheet and schedule for the first 

week of observations, (2) copies of the two observation forms which have been filled out as a model to 

follow, and (3) blank observation forms for the first week. 

2. Observation Schedule: 

♦ During the first week, you will have a 45-minute practice session in the classroom. During this 
time you should spend 15 minutes recording an ethnographic narrative (Form 1) and 30 minutes 
recording the time on task/interaction data (Form 2). 

♦ During the remaining three observation sessions you will do three 90-minute observations of the 
assigned student in the pilot/pod program. During the first 90-minute observation you will 
complete an ethnographic narrative (Form 1). During the second and third 90-minute 
observations, you will collect the descriptive data for Form 2. 

♦ During the second week, you will follow the same schedule for a student in a pull-out program. 
You should complete the forms in the same sequence and observe at roughly the same times. 

3. The Observation Forms: 

♦ Ethnographic Narrative (Form 1). As you observe the student, write a narrative account of 
what he/she is doing during each phase of the observation. Identify the classroom activity and 
describe what the student does. It is important that this narrative be in prose (in complete 
sentences) rather than in the form of sketchy notes. You and others will be reading it later. 

♦ Descriptive Observation Record (Form 2). As you observe the student, have a watch and the 
observation forms in front of you. With each classroom activity, record the beginning and ending 
times, the nature of the activity, the beginning and ending times of on-task behavior by the student, 
and a tally of the student’s interaction with the teacher (or an aide) and with other students. 

a. An “Activity" is defined as any block of time in the lesson plan. It can include 
lecturing by the teacher, pair work, worksheets, silent reading, science 
experiments, and so forth. Look at the activity as a complete unit: do not break 
it into too many smaller parts (such as “instructions”). 
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b. “Time on Task” is defined as any period of time in which the student is 
focused on the activity. If the student is focused on something else (even if 
it is productive, such as reading), he/she is “off task.” If the student finishes 
an activity early and does something else, or if he/she is in a small group but 
is not listening or speaking to the others, the student is “off task.” Obviously, 
you will need to make some judgment calls as an observer. The key is to be 
consistent in what you define as “on task” behavior. 

c. “Interaction” is defined as any time the student speaks to a teacher or aide, or 
to another student. Record this in terms of utterances. If the teacher asks the 
student three questions in a row, and the student answers each question, then you would 
record three interactions. If the student speaks in front of the class for five minutes this 
would be recorded as one interaction. (You are recording frequency, not length.) 

If the student is involved in a conversation with another student, record one interaction 
for every time he/she speaks (i.e., one for each “turn” in the conversation). 



4. Procedures. 

♦ Please introduce yourself to the teacher when you arrive in the classroom, and ask her where you 
should sit. Be unobtrusive in your seating, but position yourself so you have a good view of the 
student you are observing. If the student is repositioned in the classroom or goes to another room, 
you should follow the student in order to be within earshot. Do not sit or stand close to the 
student, however; try to keep at least 10-15 feet of distance between yourself and the student. 

♦ If possible, don’t talk with the student. (You could have a brief conversation, of course, but don’t 
make him/her overly aware of your presence.) Do not participate in classroom activities as an 
aide. If someone asks you to help a student, politely explain that you are involved in collecting 
research data. 

♦ When you enter the school, you may need to enter the main office to sign in. Follow the 
appropriate procedures for visitors to the building. 

♦ When you have completed all of the observations, make photocopies of the forms (or give them to 
Alan Seaman to photocopy) so you can keep one copy to use in your own analysis. 

5. If you have any questions about this, please feel free to call Alan Seaman at home or at work. 
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